3DeeCellTracker Demo: single mode

This notebook shows how to use 3DeeCellTracker to track cells in single mode.

The basic procedures:

1. Initialize the parameters for tracking
2. Prepare images to be tracked and the pre-trained U-Net and FFN models.
3. Optimize segmentation parameters and segment the image at volume 1.
4. Correct the segmentation at volume 1 (in other image processing software)
5. Optimize tracking parameters.
6. Track following volumes.

Please run folloing codes according to the instructions

• Some codes can be skipped if unnecessary.

Import packages

%load_ext autoreload
%autoreload 2
import os
import warnings
warnings.filterwarnings('ignore')

from IPython.core.display import display, HTML
from matplotlib.animation import FuncAnimation, ArtistAnimation
from CellTracker.tracker import Tracker, save_tracker, load_tracker
display(HTML("<style>.container { width:95% !important; }</style>"))
%matplotlib inline

Using TensorFlow backend.

1. Initialize the parameters for tracking

Image parameters

• volume: number of the volumes (time points) of the images to be tracked
• siz_xyz: size of each 3D image: (height, width, depth), unit: voxels
• z_xy_ratio: resolution (um/voxels) ratio between z (depth) and x-y plane
• z_scaling: (integer), the factor for interpolating the unbalanced z_xy resolution

Segmentation parameters

• noise_level: a value close to the averaged non-cell region intensity, which helps the program to ignore the background noises.
• min_size: the possible mimimal cell size, which helps the program to ignore small non-cell regions.
• Note: To change them after this initialization, please use ".set_segmentation()" which will delete the cached segmentation files

Tracking parameters

• beta_tk: set it higher/lower to get more coherent/independent predictions for cell positions.
• lambda_tk: set it higher/lower to get more coherent/independent predictions for cell positions.
• maxiter_tk: (integer) the number of iteration for the tracking algorithm (FFN+PR-GLS), the higher, the more accurate (but slower)

Paths

• folder_path: the path of the folder to store the data, model, and results.
  • "./xxx" indicates a folder with name "xxx" under the directory containing this jupyter notebook.
• image_name: the names of the images to be tracked.
  • "aligned_t%03i_z%03i.tif" indicates file names like: "aligned_t002_z011.tif", "aligned_t502_z101.tif", etc.
• unet_model_file: the name of the pre-trained unet model file
• ffn_model_file: the name of the pre-trained ffn model file

tracker = Tracker(
    volume_num=50, siz_xyz=(512, 1024, 21), z_xy_ratio=9.2, z_scaling=10, 
    noise_level=20, min_size=100, beta_tk=300, lambda_tk=0.1, maxiter_tk=20, 
    folder_path=os.path.abspath("./worm1"), image_name="aligned_t%03i_z%03i.tif", 
    unet_model_file="unet3_pretrained.h5", ffn_model_file="ffn_pretrained.h5")

Create folders to store data, model, and results

• /data: to store the images to be tracked
• /auto_vol1: to store the automatically generated segmentation results at volume 1
• /manual_vol1: to store the manually corrected segmentation results at volume 1
• /tracking_information: to store the coordinates information of the tracked cells
• /models: to store the pretrained and retrained models
• /unet: to store the cached predictions of cell regions
• /track_results_SingleMode: to store the tracking results (images of labels)
• /anim: to store the animation of the tracking process (used for diagnosis)
• /models/unet_weights: to store the retrained unet weights
• Note: in case one folder have been created, the program will not overwrite it.

tracker.make_folders()

Following folders were made under: /home/wen/PycharmProjects/3DeeCellTracker
worm1/data
worm1/auto_vol1
worm1/manual_vol1
worm1/track_information
worm1/models
worm1/unet
worm1/track_results_SingleMode
worm1/anim
worm1/models/unet_weights

2. Prepare images and pre-trained models

Prepare images

• Move images to the created "/data" folder. The names of the images should be consistent with the parameter "image_name"
• Move the pre-trained 3D U-Net and FFN model files to the created "/model" folder

3. Optimize segmentation parameters and segment the image at volume 1.

Modify the segmentation parameters (optional)

• Please modify these segmentation parameters only when the following segmentation results are unsatisfied
• Note: the cached segmentation in "/unet" folder will be deleted after this modification

tracker.set_segmentation(noise_level=20, min_size=100)

Segmentation parameters were not modified

Segment cells at volume 1

tracker.load_unet()
tracker.segment_vol1()

Loaded the 3D U-Net model
Load images with shape: (512, 1024, 21)
Segmented volume 1 and saved it

Draw the results of segmentation (Max projection)

anim_seg = tracker.draw_segresult(percentile_high=99.8)

Segmentation results (max projection):

Show segmentation in each layer

HTML(anim_seg)

Once Loop Reflect

4. Manually correct the segmentation at volume 1 and load it.

Manual correction

• Delete non-cell regions;
• Add cells not labeled;
• correct cell boundaries in other software (such as ITK-SNAP)

Move files to the folder

• save the corrected segmentation as 2D images (with different z)
• Move the corrected segmentation to the created "/manual_vol1" folder

Load the manually corrected segmentation

tracker.load_manual_seg()

Loaded manual _segment at vol 1

Re-train the U-Net using the manual segmentation (optional)

• Please do this only if the prediction by U-Net is poor (e.g. many cells were not ignored or many non-cell regions were identified as cells).

tracker.retrain_unet()

Images were normalized
Images were divided
Data for training 3D U-Net were prepared
WARNING:tensorflow:From /home/wen/anaconda3/envs/3DCT/lib/python3.7/site-packages/keras/optimizers.py:790: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead.

WARNING:tensorflow:From /home/wen/anaconda3/envs/3DCT/lib/python3.7/site-packages/tensorflow/python/ops/nn_impl.py:180: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
144/144 [==============================] - 9s 61ms/step
val_loss before retraining:  0.017810788574732013

Epoch 1/1
60/60 [==============================] - 72s 1s/step - loss: 0.0183 - val_loss: 0.0316
Epoch 1/1
60/60 [==============================] - 68s 1s/step - loss: 0.0146 - val_loss: 0.0178
val_loss updated from  0.017810788574732013  to  [0.017790769653705258]

Epoch 1/1
60/60 [==============================] - 69s 1s/step - loss: 0.0133 - val_loss: 0.0283
Epoch 1/1
60/60 [==============================] - 67s 1s/step - loss: 0.0133 - val_loss: 0.0219
Epoch 1/1
60/60 [==============================] - 68s 1s/step - loss: 0.0125 - val_loss: 0.0346
Epoch 1/1
60/60 [==============================] - 68s 1s/step - loss: 0.0133 - val_loss: 0.0186
Epoch 1/1
60/60 [==============================] - 68s 1s/step - loss: 0.0114 - val_loss: 0.0180
Epoch 1/1
60/60 [==============================] - 69s 1s/step - loss: 0.0127 - val_loss: 0.0166
val_loss updated from  [0.017790769653705258]  to  [0.016596044879002472]

Epoch 1/1
60/60 [==============================] - 67s 1s/step - loss: 0.0125 - val_loss: 0.0213
Epoch 1/1
60/60 [==============================] - 69s 1s/step - loss: 0.0147 - val_loss: 0.0178

• Set "step" with the satisfied segmentation result
• If the initial segmentation is better, set "step=0"

tracker._select_weights(step=8)
tracker.set_segmentation(noise_level=20, min_size=100, reset_=True)
tracker.segment_vol1()
anim_seg = tracker.draw_segresult(percentile_high=99.8)

Segmentation parameters were not modified
All files under /unet folder were deleted
Load images with shape: (512, 1024, 21)
Segmented volume 1 and saved it
Segmentation results (max projection):

Interpolate cells to make more accurate/smooth cell boundary

tracker.interpolate_seg()
tracker.draw_manual_seg1()

Interpolating... cell:164

Initiate variables required for tracking

• cal_subregions(): required by "accurate correction" in following procedures
• check_multicells(): if multiple separated cells were labeled as the same one, generate an error message.

tracker.cal_subregions()
tracker.check_multicells()
tracker.load_ffn()
tracker.initiate_tracking()

Calculating subregions... cell: 164
Checked mistakes of multiple cells as one: Correct!
Loaded the FFN model
Initiated coordinates for tracking (from vol 1)

5. Optimize tracking parameters.

Modify tracking parameters if the test result is not satisfied (optional)

tracker.set_tracking(beta_tk=300, lambda_tk=0.1, maxiter_tk=20)

Tracking parameters were not modified

Test a matching between volume 1 and a target volume, and show the FFN + PR-GLS process by an animation (5 iterations)

• Red circles: manually corrected cells in vol 1
• Blue cross: cells detected by 3D U-Net in target volume
• Blue arrows: predicted transformation (cell movements)

anim_tracking, results = tracker.match(target_volume=50)
HTML(anim_tracking)

Matching between vol 1 and vol 50 was computed

Once Loop Reflect

Show the accurate correction after the FFN + PR-GLS transformation

• Arrows show the correction of the cell positions
• Cells without arrow: no correction.

tracker.draw_correction(*results[2:])

Show the superimposed cells + labels before/after tracking

tracker.draw_overlapping(*results[:3])

6. Tracking following volumes

Track and show the processes

%matplotlib notebook
from_volume=2
ax, fig = tracker.track_animation(from_volume)
tracker.track_and_confirm(from_volume, fig, ax)

Show the processes as an animation (for diagnosis)

%matplotlib inline
track_anim = tracker.track_animation_replay(from_volume=2)
HTML(track_anim)

Once Loop Reflect